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Revisit of KVM networking with linux bridge 


T 
router: 10.0.0.1 
(def. gatewhy for VMs) 


nova-compute host =~ - ~__ _ EM | 3 —--- nova-compute host 
nici switch rali 
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Single-host networking 


e KVM host == nova-compute 
e router == nova-network 


nova-network host 
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What happens if nova- nost dies 


VM:-root$ ping google.com 


No route to host 


linux 
bridge 
nova-compute host ^———- ---... DH 3X3 3b -—--- nova-compute host 
---4] switch þ-- 
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Multi-host networking C» 


Move routing from the central server 
to each compute node independently to eth1 


prevent SPOF. routing/ NAT 
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Multi-host networking 


Compute servers maintain Internet access independent from each other. Each of 
them runs nova-network & nova-compute components. 


public ip ^^ — ` 47 ===, public ip 


__, eth ae 
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! routing/NAT i 
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nova-compute & | “= ~~ LL EMEN «wh ——--- nova-compute & 
nova-network host nn switch ~ nova-network host 
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Multi-host networking - features 


e Independent traffic: 
Instances on both compute nodes access external 
networks independently. For each of them, the local linux 
bridge is used as default gateway 

e Routing: 
Kernel routing tables are checked to decide if the packet 
should be NAT-ed to eth1 or sent via ethO 


e |P address management: 
nova-network maintains IP assignments to linux bridges 
and instances in nova database 
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Multi-host networking - nova.conf 
e nova.conf: 


O --multi host=True/False: determine whether openstack should run 
nova-network daemon on each compute node or just on one, dedicated 
server 


O --public_interface=eth1: interface to be used as the one facing external 
networks 
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Network manager 


e Determines network layout of the cloud 
infrastructure 


e Capabilities of network managers 


MIRANTIS 


O 


O O O O O ©. 0 


Plugging instances into linux bridges 
Creating linux bridges 

IP allocation to instances 

Injecting network configuration into instances 
Providing DHCP services for instances 
Configuring VLANs 

Traffic filtering 

Providing external connectivity to instances 
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FlatManager 
e Features: 


O Operates on ONE large IP pool. Chunks of it are shared between tenants. 
O Allocates IPs to instances (in nova database) as they are created 

O Plugs instances into a predefined bridge 

O Injects network config to /etc/network/interfaces 


4 


`A 4 `A 
; eth1 |  eth1 | 
Te eee ns d 
VM 
VM letc/network/interfaces: 
"address 10.0.0.2 
gateway 10.0.0.1" 
linux linux 
bridge bridge 
10.0.0.1(gw) 
ethO ethO 
nova-compute & nova-compute & 
nova-network nova-network 
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FlatManager - nova.conf 
e nova.conf: 


O --network manager=nova.network.manager.FlatManager 
O --fixed range=10.0.0.0/8: ip pool to be shared among the tenants 
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FlatDHCPManager 


e Features: 


O Operates on ONE large IP pool. Chunks of it are shared between tenants 
O )Allocates IPs to instances (in nova database) as they are created 

O Creates a bridge and plugs instances into it 
O 


Runs a DHCP server (dnsmasq) for instances to boot from 


á ^ LE: 
Gili mr 
L----1 L — LALII. 
VM 
obtain dhcp static lease: 
VM ip: 10.0.0.2 
gw: 10.0.0.1 


linux 
«e»... 


nova-compute 
& nova-network 


nova-compute & 
ova-network 
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DHCP server (dnsmasq) operation 

e is managed by nova-network component 

e in multi-host networking runs on every compute node and 
provides addresses only to instances on that node (based 
on DHCP reservations) 
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ip: 10.0.0.6 
gw: 10.0.0.3 gw: 10.0.0.3 


dnsmasą: static leases for 
10.0.0.2 & 10.0.0.5 


dnsmasą: static leases for 
10.0.0.4 & 10.0.0.6 


nova-compute & = nova-compute De 
nova-network =a & nova-network- ~ 


^*| switch | 
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FlatDHCPManager - nova.conf 
e nova.conf: 


O 
O 
O 


MIRANTIS 


--network managerznova.network.manager.FlatDHCPManager 
--fixed range=10.0.0.0/8: ip pool to be shared among the tenants 


--flat network bridge=br100: bridge to which instances will be attached 
(FlatDHCPManager creates it in case it is not present) 


--flat network dhcp start=10.0.0.2: start allocating ip addresses to 
instances via dhcp with this address 


--allow same net traffic-true/false: determines whether instances 
belonging to different tenants can access one another by default 
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VlanManager 
e Features: 


O Can manage many IP subnets 

Uses a dedicated network bridge for each network 

Can map networks to tenants 

Runs a DHCP server (dnsmasq) for instances to boot from 


Separate ^^*work traffic with 802.10 VLANs 
i eth1 | 


 —— | 


VM net1 VM net2 


O 
O 
O 
O 


dnsmasq net1 7 dnsmasq net2 
STÐ 


nova-compute 
8. nova-network 
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nova-compute & 
ova-network 


VlanManager - switch requirements 
The switch requires support for 802.10 tagged VLANs to 
connect instances on different compute nodes . 

| eth1 | ‘eth | 


Á — mn Á —— um 


VM neti VM net2 VM net2 VM net1 


eth0.1 oo eo 00 


nova-compute nova-compute 
& nova-network & nova-network 


tagged traffic 


802.1Q capable switch 
MIRANTIS © Mirantis, Inc, 2012. All rights reserved. 


VlanManager - nova.conf 


e nova.conf: 
O 


--network_manager=nova.network.manager.VlanManager 
O 


--vlan_interface=eth0: interface to which bridges and VLANs are 
attached 


O --vlan start=100: first VLAN for tenants networks 
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Network managers comparison 


Name ^ Possible use cases Limitations 


FlatManager Deprecated - should not be used for Only Debian derivatives 
any deployments. supported. 
Single IP network suffers from 
scalability limitations. 


FlatDHCPManager Internal, relatively small corporate Instances share the same linux 
clouds which do not require tenant bridge regardless which tenant 
isolation. they belong to. 

Limited scalability because of one 
huge IP subnet. 


VlanManager Public clouds which reguire L2 traffic Reguires 802.10 capable switch. 
isolation between tenants and use of Scalable only to 4096 VLANs 
dedicated subnets. 


m 
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Inter-tenant traffic 


Compute node's routing table 
consulted to route traffic 
between tenants' networks 
(based on IPs of the linux 
bridges) 
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public ipf " 


routing I 
0.100.0.0 via br100 į 
0.200.0.0 via br200 i 


=== = = SR umo == == = = om 9 


nova-compute 
& nova-network 
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Accessing internet 


e eth1 address is set as the 
compute node's default gateway | 

e Compute nodes routing table 
consulted to route traffic from 
the instance to the internet over | 
the public interface (eth1) FIIO EE 

€ source NAT is performed to the pac RUE DE 
compute node's public address 


public ipf g 


nova-compute 
8. nova-network 
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Accessing internet - nova.conf 
e nova.conf: 


O --routing source ip=<public ip»: Public IP of network host. When 
instances without a floating IP hit the Internet, traffic is SNAT-ed to this 
IP address 
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Floating & fixed IPs 
e Fixed IPs: 


O 


O 
O 


O 


given to each instance on boot 

private IP ranges (10.0.0.0, 192.168.0.0, etc.) 

only for communication between instances and to 
external networks 

inaccessible from external networks 


e Floating IPs: 


MIRANTIS 


O 


O 


allocated & associated to instances by cloud users 
bunches of publicly routable IPs registered in 
Openstack by cloud dmin 


o accessible from external networks 


multiple floating IP pools, leading to different ISP-s 
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Floating IPs 


e User associates a floating 


IP with VM: 
o floating IP is added as a 


floating IP 
added as a 

secondary IP 
on eth1 


vm float ip: 92.93.94.95 


secondary IP address on usc p ad 

compute node's eth1 (public / ,----- AA o 
floating IP DNAT: 

| F) -d 92.93.94.95/32 -j DNAT -- 


to-destination 10.0.0.2 


o DNAT rule is set to redirect 
floating IP -> fixed IP 
(10.0.0.2) 


nova-compute & 
nova-network host 
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Floating IPs - nova.conf 
e nova.conf: 


O --auto assign floating ip-true/false: Enables auto-assignment of IP 
addresses to VMs 
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Openstack networking theory - recap 


e What SPOF is addressed with multi-host 
networking 

e Which network managers are able to configure 
DHCP 

e What feature is a must-have for the switch when 
VlanManager is used 

e What NAT rule is set when a floating IP is 
attached to an instance 

e What are key limitations of VlanManager 
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Openstack networking - network flows 


We present all the scenarios of network 
communications that can happen to instances. 


Key assumptions: 


e multi-host networking 
e VlanManager 
e two tenants with separate networks 
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Flow 1: instance boot 


tenant2 


eth1 
91.207.15.106 


S | L 7 
-— eth0 D 


œS 
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Instance Boot steps 


1. VM 1 boots and sends DHCPDISCOVER broadcast 
message to local network 

2. Message got broadcasted over br100 

3. dnsmasq answers with DHCPOFFER 
a. IP address: 10.0.0.5 
b. Default Gateway: 10.0.0.1 (br100) 

4. Same for VM 4 (IP: 10.0.0.6, GW: 10.0.0.5) 


Instances running on different hosts have different 
default gateways 
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Flow 2: instance connects to the world 


eV Í 
etht etht 
91.207.15.105 91.207.15.106 


tenant2 


S | 
— 


eth0 = 


switch 


SS 
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Connect to google.com steps 


MIRANTIS 


. VM 1 request 8.8.8.8 (Google's public DNS) - 


8.8.8.8 Is not in 10.0.0.0/24 

VM 1 sends data to default gateway (br100) 
nova-network sees that 8.8.8.8 doens t directly 
connected, so it sends the packet to compute 
node default gateway (eth1) 


. eth1 maintains NAT between all tenant instances 


and public internet 
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Flow 5: Instances of one tenant communicate 
e compute node 


(priv) viant02 (priv) vian102| 


switch 
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VM-to-VM on the same host steps 


— 


. VM 1 doesnt know MAC address of VM 2 yet 

2. VM 1 sends ARP broadcast packet 

3. br100 broadcasts message to the whole tenant 
network 

4. Once VM 2 MAC address is determined, IP 

packets are sent to it from VM 1 
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Flow 4: Instances of one tenant communicate 


VM 6 
10.1.0.4 


br102 
10.1.0.5 
lm». 


switch 
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VM-to-VM on different hosts steps 


— 


. VM 1 doesnt know MAC address of VM 5 yet 

2. VM 1 sends ARP broadcast packet 

3. br100 broadcasts message to the whole tenant 
network 

4. The packet is tagged with vlan100 tag and sent 
to the switch (should be in "trunk" mode) 

5. Switch broadcasts packet to all connected 
bridges 

6. Packet is being untagged and sent to VM 5 

through br100 


MIRANTIS © Mirantis, Inc, 2012. All rights reserved. 


Flow 5: instances of different tenants 
communicate within the same compute node 


eth1 
91.207.15.106 


tenant2 


eth1 
91.207.15.105 


tenant2 


outing 
SE 


switch 
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VM-to-VM on same host (different tenants) 


1. VM 3 is not in 10.0.0.0/24 so VM 1 sends packet 
to default gateway (br100) 

2. br100 sends packet to nova-network (router), 
which forwards it to br102 (based on routing 
table) 

3. br102 determines VM 5 MAC based on ARP 
protocol 

4. VM 3 replies back to VM 1 
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Flow 6: instances of different tenants 


communicate within different compute nodes 
| 


tenant2 
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Flow 7: Why you should avoid communication on 
fixed IPs between tenants (and use floating) 
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Flow 7: ...continued 


VM 1 float: 
91.208.23.11 (4) 
etht: 
91.207.15.105 


VM 6 float: 
91.208.23.16 
(5) eth: 
91.207.15.105 


m 
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